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The meta-inference in mixed methods research 


A genuine mixed methods study culminates in a meta-inference, a conclusion that connects or inte- 
grates various claims, some resulting from the analysis of qualitative data, others from the analysis 
of quantitative data (Creamer, 2018, p. 110; Teddlie & Tashakkori, 2009, p. 152). As an example, 
López and Tashakkori (2006) compared the effects of two types of bilingual education programs 
(two-way, transitional) on the attitudes and academic achievement of fifth-grade students. Their 
meta-inference states that “two-way and transitional programs have differential effects, with the 
two-way programs having a more positive effect on the pace at which oral language is acquired, 
proficiency in Spanish, and attitudes toward bilingualism” (Teddlie & Tashakkori, 2009, p. 268). 

The meta-inference is the final step in the mixed methods research process (Figure 4.1). It is 
the result of two successive phases. In Phase 1, claims are obtained through a process called mixing 
methods, in which qualitative data are collected and analyzed and quantitative data are collected 
and analyzed. Applying the separate qualitative and quantitative methods leads to separate quali- 
tative findings and quantitative findings, and these separate findings, in turn, lead to separate con- 
clusions or claims. The application of methods and obtaining of findings may proceed in various 
cycles. Quantitative methods may be applied on the basis of qualitative findings and vice versa, 
as indicated by the arrows between findings and methods in Figure 4.1 (Creswell & Plano Clark, 
2018, p. 66). In Phase 2, claims obtained through qualitative research methods are connected to 


-C 
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Phase 1: Phase 2: 
Obtaining Claims Integrating Claims 


Figure 4.1 Mixing Methods and the Meta-Inference in the Mixed Methods Literature 
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claims obtained through quantitative methods — that is, claims from both types of methods are 
integrated. The result is a meta-inference, such as the one cited in the previous paragraph. 

Through the integration of their claims in the meta-inference, the qualitative and quanti- 
tative methods become mutually illuminating (Bryman, 2007). However, as Bryman noted, in 
many studies intended as mixed methods studies, no actual mixing of methods or findings — and 
no integration of claims — takes place; the gray, integrating elements of Figure 4.1 are missing. 
As a result, there is no meta-inference. Such a study is not considered a real mixed methods 
study but rather two separate studies, one qualitative and one quantitative (the two separate 
research strands without connections in Figure 4.1). 

Not surprisingly, the meta-inference is a highly valued outcome in mixed methods research 
(Uprichard & Dawney, 2019); it has been called the “hallmark” of mixed methods research 
(Moseholm & Fetters, 2017), it occurs “at the heart of mixed methods research” (Bazeley, 2018, 
p. 9; see also Tunarosa & Glynn, 2017), and “the actual dialectic mixing of consequence lies in 
the construction or composition of inferences, drawn from purposeful conversations of different 
threads of data patterns” (Greene & Hall, 2010, p. 125). The meta-inference is an indicator of a 
mixed methods study’s yield (O’Cathain et al., 2010) and quality. Some mixed methods scholars 
go even further by claiming that “the quality of the entire research project depends on the degree 
to which integration, blending, or linking of QUAL and QUAN inferences is achieved” (Ted- 
dlie & Tashakkori, 2009, p. 292; emphasis added). Thus, it is not the mixing of methods but the 
integration of inferences or claims that is ultimately important. 

Paradoxically, however, the integration of claims has received very little attention in the 
mixed methods literature. When mixed methods scholars discuss integration, they are almost 
always referring to mixing methods, often called integration of methods, not integrating claims. 
In other words, they focus on the obtaining claims part of Figure 4.1. Bryman’s (2006) seminal 
article, for example, explains which methods are often used in mixed methods research, what 
purposes for combining these methods can be distinguished, and how often these were used in 
a representative sample of mixed methods studies. More recently, Maxwell et al. (2015) discuss 
“how to integrate qualitative and quantitative methods and data” (p. 227) and conduct studies that 
“provide valuable insights into the strategies, outcomes, and difficulties of integrating qualitative 
and quantitative methods, data, and mental models” (p. 228; emphasis added). Fetters et al. (2013) 
distinguish between integration at the study design level, methods level, and interpretation and 
reporting level. What is integrated at these levels are qualitative and quantitative research procedures 
and data (p. 2135), as well as findings or results (p. 2142). 

One notable exception to this trend of ignoring claims is philosopher Nancy Cartwright, 
who has repeatedly pointed out the importance of paying attention to claims in designing 
mixed methods research (Cartwright, 2018; Cartwright & Hardie, 2012). Because a conclusion 
cannot be stronger than the premises on which it rests, researchers designing mixed methods 
research should examine the claims, sub-claims, and sub-sub-claims needed to support their 
main claim and should examine what evidence is needed to warrant these claims: "First mix the 
claims, then mix the methods” (Cartwright, 2018). 

Given that the meta-inference is the most important outcome of mixed methods research, 
there is a knowledge gap. The mixed methods literature does not contain a discussion of the 
internal claim structure of meta-inferences, and there are no descriptions of how integration 
of which claims leads to the meta-inference in actual mixed methods research. As a result, the 
mixed methods literature provides little guidance for researchers on how to develop a meta- 
inference from actual data. 

This chapter seeks to fill this gap. Using three real-life examples of mixed methods research, 
it will show how the meta-inference and its internal structure are developed in successive steps of 


56 


Developing the mixed methods meta-inference 


' 
I 

Method a Finding A ; Claim [1] 
L 


Methodb Y —,———— 0 c77-7---------- lļ----=======--- 
Finding C : 
` 


1 
[ 
1 

ton a CÓ u€ CE m Er ee s o E m ————Ó 
' 
' 


Finding D Claim [6] 
LI 


Integrated Claim [7] 
= Meta-Inference 


Figure 4.2 Emergent Development of the Meta-Inference in This Chapter 


claim integration (Figure 4.2). It defines one such instance of claim integration as “one occasion 
during the research process at which two or more simpler claims are integrated into one more 
complex claim." 

Furthermore, this chapter will show that claim integration is not unique to mixed methods 
research but is visible in mono-method research as well. Finally, it discusses the general pattern 
of successive claim integration visible in the three real-life examples and describes the conse- 
quences for a view on the quality of mixed methods research. 


Integration of claims in three real-life examples of mixed methods 
research 


Example 1 


The first real-life example is a study by Assen et al. (2016) investigating the influence of teacher 
beliefs on teacher behavior among so-called tutors, teachers at one Dutch university whose 
task it was to facilitate and to support the students’ learning process using a student-oriented 
approach, rather than a more traditional teacher-centered style of knowledge transmission. 
Their abstract reads as follows: 


The purpose of this study was to explore the discrepancy between teacher beliefs and 
behavior in a Problem-Based Learning (PBL) environment. Using a survey and obser- 
vations, this study demonstrated that tutors prefer learner-oriented beliefs, but in their 
teacher behavior they showed a more traditional approach to teaching. Analysis of 
semi-structured interviews indicated that this inconsistency could be attributed to the 
way in which problem-based learning is embedded in the curriculum, the confidence 
teachers have in the self-directed capabilities of students, and the self-confidence of 
teachers regarding their own facilitation skills. 


(p. 12) 


The claim structure of Example 1 is presented in Table 4.1. 

Table 4.1 shows how successive integration of claims leads to a meta-inference. The final 
Integrated Claim (henceforth: IC) [8] is the meta-inference, as indicated by its heading. It is 
the result of a three-step process in which Claims [1]-[7] are integrated. Each instance of claim 
integration is displayed between horizontal lines. Instance of Integration I (henceforth: I) inte- 
grates Claims [1] and [2] into IC [3]. [1] had been derived from findings obtained through a 
survey, a quantitative method, whereas [2] had been derived from findings obtained through 
observations, a qualitative method. Accordingly, I bears the label mixed methods because it is an 
instance of integration through mixing methods. III is an instance of mixing of methods as well. 
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Table 4.1 Development of the Meta-Inference in Example 1 


Obtaining Claims Integrating Claims 
Methods Claims Integrated Claims Connection 
I. Mixed methods 
Survey — [1] Tutors prefer learner- IC [3] Tutors prefer learner- Contradiction 
oriented beliefs. } oriented beliefs, but they show 
Observations — [2] Tutors show a more a more traditional approach to 
traditional approach to teaching. 
teaching. 
II. Mono-method (qualitative) 
Semi- — [4] Problem-based learning is IC [7] Problem-based learning is ^ Juxtaposition 
structured embedded in the curriculum embedded in the curriculum 
interviews in a specific way. in a specific way, and teachers 
— [5] Teachers have a certain have a certain amount of 
amount of confidence in the } confidence in the self-directed 
self-directed capabilities of capabilities of students and 
students. showed varying amounts of 
— [6] Teachers showed varying self-confidence regarding their 
amounts of self-confidence own facilitation skills. 
regarding their own 
facilitation skills. 
III. Mixed methods Meta-Inference 
Survey & — IC [3] Tutors prefer learner- IC [8] Tutors prefer learner- Explanation 
Observations oriented beliefs, but they oriented beliefs, but they show 
show a more traditional a more traditional approach to 
approach to teaching. } teaching because problem- 
Semi- — IC [7] Problem-based based learning is embedded in 
structured learning is embedded in the the curriculum in a specific 
interviews curriculum in a specific way way and teachers have a certain 
and teachers have a certain amount of confidence in the 
amount of confidence in the self-directed capabilities of 
self-directed capabilities of students and showed varying 
students and showed varying amounts of self-confidence 
amounts of self-confidence regarding their own facilitation 
regarding their own skills. 


facilitation skills. 


Note: IC = integrated claim. 


It combines IC [3], derived from a quantitative survey and qualitative observations, and IC [7], 


derived from a qualitative semi-structured interview. 


Perhaps surprisingly within a mixed methods study, II is an instance of mono-method integra- 
tion: Claims [4], [5], and [6] are derived from the same data source, a semi-structured interview. 
As IC [7], these three claims together explain the contradiction of IC [3]. Each of them alone 
cannot explain the contradiction, but together they can. This means that they have to be inte- 


grated before they can form an explanation, which is done in II. 


Integrated claims that are the result of integration earlier in the research process are often 
themselves integrated with other claims later in the process, so that, in the end, the meta-infer- 
ence contains various integrated claims. In Table 4.1, IC [3] and IC [7] are integrated in III to 
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form IC [8], the meta-inference. In Table 4.1, the meta-inference IC [8] contains all claims that 
had been derived from the separate sources, namely [1], [2], [4], [5], and [6]. 

The connections among the claims within one instance of integration are described in the 
last column of Table 4.1 as contradiction, juxtaposition, and explanation, respectively. Thus, 
in IC [3], “[1] tutors prefer learner-oriented beliefs” is a contradiction of “[2] in their teacher 
behavior, they showed a more traditional approach to teaching,” indicated in Example 1’s abstract 
with the word but; in IC [7], “[4] the way in which problem-based learning is embedded in 
the curriculum,” “[5] the confidence teachers have in the self-directed capabilities of students,” 
and “[6] the self-confidence of teachers regarding their own facilitation skills” are juxtaposed, 
indicated by commas and and; and in IC [8], IC [7] is an explanation for the contradiction of 
IC [3] (“this inconsistency could be attributed to the way in which . . .”). 


Example 2 


A study by Clark and Moss (Clark, 2005) accompanied the redevelopment of the outdoor 
environment of a preschool in the United Kingdom. Their study involved 28 three- to four- 
year-olds, their parents, and preschool practitioners and managers. It answered two interrelated 
research questions: Which places do children see as important in this outdoor space? How do 
the children use these places? The children were involved in data collection and took photo- 
graphs of important objects. One of the objects was the playhouse. According to Clark (2005): 


Observing the children revealed the house to be a key resource for them. The children 
confirmed this through their photographs, the tour and their interviews. Parents also 
mentioned the house as an important space in the preschool. However, the interviews 
with practitioners showed that the house was a source of tension. They felt it was 
too small. The review with children, practitioners, and Learning through Landscapes 
recognised these opposing views and raised some possible solutions. The preschool 
has now turfed a new area for children to use to build their own temporary structures. 


(p. 16) 


The claim structure of Example 2 is displayed in Table 4.2. 

Table 4.2 identifies six instances of claim integration, of which II, III, IV, and VI are instances 
of mixed methods integration. V involves mono-method integration: practitioners mentioned 
both [10] the playhouse as a source of tension and [12] their explanation (“the playhouse was 
too small”) in their interviews, perhaps even in the same sentence. Despite its narrow evidence 
base, IC [13] (“the playhouse is a source of tension, because it is too small”) nevertheless 
played an important role in the further development of the preschool environment: according 
to Example 2’s text, the review “recognised these opposing views,” and the preschool “turfed a 
new area for children to use to build their own temporary structures.” 

A third form of integration is seen in I. Here, four claims derived from various qualitative 
methods are integrated: observations, photographs, tours, and interviews. This form of integra- 
tion is called multimethod integration, the integration of claims derived from multiple methods 
that are of the same kind (i.e., either all qualitative methods or all quantitative methods). 

Instances of integration II, III, IV, and VI each include one claim that is an integrated claim 
formed in a previous step. The labeling of each as a mixed methods or mono-method instance 
of claim integration depends on the variety of methods used in all previous instances of integra- 
tion that lead up to the integrated claim. Thus, a meta-inference in mixed methods research 
will bear the label “mixed methods" because it involves at least one instance of mixed methods 
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Table 4.2 Development of the Meta-Inference in Example 2 


Obtaining Claims 


Integrating Claims 


Methods 


Obtained Claims 


Integrated Claims 


Connection 


I. Multimethod (qualitative) 


Observations 


Contents of children's 
photographs 
Children’s tours 


Interviews with children 


II. Mixed methods 

[Various qualitative 
methods] 

Counts of children’s 
photographs 

III. Mixed methods 

[Various qualitative 
methods and one 
quantitative method] 

Interviews with parents 


IV. Mixed methods 

[Various qualitative 
methods and one 
quantitative method] 

Interviews with 
practitioners 


— 


— 


=> 


[1] The playhouse is a key 
resource for the children. 
[2] The playhouse is a key 
resource for the children. 
[3] The playhouse is a key 
resource for the children. 
[4] The playhouse is a key 
resource for the children. 


IC [5] The playhouse is a key 
resource for the children. 

[6] The playhouse is a key 
resource for the children. 


IC [7] The playhouse is a key 
resource for the children. 


[8] The playhouse is an important 
space in the preschool. 


IC [9] The playhouse is a key 
resource for the children. 


[10] The playhouse is a source of 
tension. 


V. Mono-method (qualitative) 


Interviews with 
practitioners 


VI. Mixed methods 

[Various qualitative 
methods and one 
quantitative method] 

Interviews with 
practitioners 


— 


=> 


— 


— 


Note: IC — integrated claim. 


[10] The playhouse is a source of 
tension. 
[12] The playhouse is too small. 


IC [9] The playhouse is a key 
resource for the children. 


IC [13] The playhouse is a source 


of tension because it is too 
small. 
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IC [5] The 
playhouse is 

a key resource 

for the children. 


IC [7] The 
playhouse is 

a key resource 

for the children. 


IC [9] The 
playhouse is 

a key resource 

for the children. 


IC [11] The 
playhouse is 

a key resource 

for the children, 

but it is also 

a source of tension. 


IC [13] The 
playhouse is 

a source of tension 

because it is too 
small. 

Meta-inference 

IC [14] The 
playhouse is 

a key resource 

for the children, 
but it is also a 
source of tension 

because it is too 
small. 


Confirmation 


Confirmation 


Confirmation 


Contradiction 


Explanation 


Explanation 


Developing the mixed methods meta-inference 


integration during its development process, but, as Tables 4.1 and 4.2 have shown, not all 
instances of integration leading up to the meta-inference involve mixing methods. 

In IC [9], Claims [2], [3], [4], [6], and [8] all confirm [1]: “The playhouse is a key resource 
for the children.” In the absence of exact knowledge of the research process, Table 4.2 assumes 
that confirmation took place through three separate instances of claim integration (I, II, and 
III), based on the idea that in mixed methods research, evidence is commonly added step by 
step and thus integrated claims also develop step by step (Figure 4.2). The route to IC [9] may 
actually have been slightly different from Table 4.2. Given the various sources of qualitative data, 
however, the route almost certainly included several instances of integration that integrated two 
or more claims derived from multiple qualitative methods. In other words, it almost certainly 
contained multimethod instances of integration in addition to mixed methods integration. 


Example 3 


Glewwe et al. (2009) studied the effects of providing textbooks to schoolchildren in rural 
Kenya, in schools where textbooks had not been used before. They compared test scores of 
children in the 50 intervention schools with those of a control group, which showed no effect. 
In addition, the researchers went to each school and asked a child with a median score from 
each class to read his or her textbook aloud and to answer a few questions. Further subgroup 
analysis of the test scores was used to determine the differential effect for children with different 
pretest scores. According to their abstract: 


A randomized evaluation in rural Kenya finds, contrary to the previous literature, 
that providing textbooks did not raise average test scores. Textbooks did increase the 
scores of the best students (those with high pretest scores) but had little effect on other 
students. Textbooks are written in English, most students’ third language, and many 
students could not use them effectively. 


(p. 112) 


The claim structure of Example 3 is displayed in Table 4.3. 

Table 4.3 shows two additional forms of mono-method integration: within qualitative 
observations (II) and through quantitative subgroup analysis (II). Mono-method integration 
through quantitative subgroup analysis is a form of integration that has not yet been discussed in 
the mixed methods literature. It occurs when quantitative findings for one subgroup support a 
claim that is the opposite of the claim for the other subgroup, in this case Claim [7], “textbooks 
did increase text scores” for the children with high pretest scores, and Claim [8], “textbooks did 
not increase text scores” for the other children. 

In addition, Table 4.3 displays two instances of integration involving claims not based on data 
collection within the study, called Previous research [1] and Reasoning [11] and [12]. Instance 
of integration I involves Claim [1] from previous research, “providing textbooks raises aver- 
age test scores in primary schools.” A contradiction is noted between [1] and the quantitative 
findings of the study, which show that [2] “providing textbooks did not raise average scores in 
primary schools in rural Kenya.” 

Reasoning in [11] and [12] is a shorthand for “reasoning on grounds other than collected 
data.” Thus, in V, labeled Subgroup analysis & reasoning, an explanation for [7] “providing text- 
books did have an effect for students with high pretest scores” is found through reasoning 
without data collection. Drawing from their general knowledge, the researchers assumed that 
[11] textbooks can only have an effect for those who are able to read them, and from that, they 
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Table 4.3 Development of the Meta-Inference in Example 3 


Obtaining Claims Integrating Claims 


Methods Obtained Claims 


Integrated Claims 


Connection 


I. Previous research and quantitative methods 


Previous research — [1] Providing textbooks IC [3] Providing textbooks Contradiction 
raises average test scores in raises average test scores 
primary schools. } elsewhere, but it does 
Quantitative — [2] Providing textbooks not do so in primary 
methods does not raise average schools in rural Kenya. 
test scores for children in 
primary schools in rural 
Kenya. 
II. Mono-method (qualitative) 
Observations — [4] Most children could not IC [6] Most children could Explanation 
read their textbook. } not read its textbook 
Observations — [5] The textbook is in because it is written in 
English, which is not the English, which is not 
native language of the their native language. 
children. 
III. Mono-method (quantitative) 
Subgroup analysis — [7] Textbooks did increase IC [9] Textbooks did Contradiction 
the scores of the students increase the scores of 
with high pretest scores. } the students with high 
Subgroup analysis — [8] Textbooks did not pretest scores but not 
increase the scores of the those of students with 
students with intermediate intermediate or low 
or low pretest scores. pretest scores. 
IV. Mixed methods 
Quantitative — [8] Textbooks did not IC [10] Textbooks did not Explanation 
subgroup increase the scores of the have an effect for most 
analysis students with intermediate children because they 
or low pretest scores. } could not read their 
Observations — IC [6] Most children could textbooks because these 
not read their textbook were written in English, 
because it is written in which is not their native 
English, which is not their language. 
native language. 
V. Quantitative subgroup analysis & reasoning 
Subgroup analysis — [7] Textbooks did increase IC [13] Textbooks did Explanation 
the scores of the students increase the scores of the 
with high pretest scores. } students with high pretest 
Reasoning — [11] Textbooks can only scores because these 
have an effect for children children were able to read 
who can read them. } them. 
Reasoning — [12] Most likely, students 
with high pretest scores 
were able to read their 
textbooks. 
(Continued) 
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Table 4.3 (Continued) 


Obtaining Claims 


Integrating Claims 


Methods Obtained Claims Integrated Claims Connection 
VI. Various strategies, including mixed Meta-inference 
methods 
Previous — IC [3] Providing textbooks IC [14] Providing textbooks Explanation 


research & raises average test scores 


quantitative elsewhere, but it does not 
methods do so in primary schools 
in rural Kenya. 

Mixed methods = — IC [10] Textbooks did not 
have an effect for most 
children because they 
could not read them 
because they were written 
in English, which is not 
their native language. 

Subgroup — IC [13] Textbooks did 

analysis & increase the scores of the 
reasoning students with high pretest 


scores because these 
children were able to read 
them. 


raises average test scores 
elsewhere, but it does 
not do so in primary 
schools in rural Kenya. 
Textbooks did not have 
an effect for most children 
because they could not 
read them because they 
were written in English, 
which is not their native 
language. However, 
textbooks did increase 
the scores of the students 
with high pretest scores, 
because these children 
were most likely able to 
read them. 


Note: IC = integrated claim. 


deduced that [12] “children with high pretest scores must have the ability to read their text- 
books.” The derivation of both [11] and [12] did not involve data collection. 

VI, which results in the meta-inference, bears the label Various strategies, including mixed meth- 
ods. Because IC [3] and IC [13] involved the integration of claims that had not been derived 
from applying methods, VI, which integrates IC [3] and IC [13], also involves more than mix- 
ing methods. 


Comprehensive inferences in mono-method research 


Apparently, mixed methods studies also contain multimethod and mono-method integration, 
which raises the question of whether the meta-inference, this hallmark of mixed methods 
research, is truly unique to mixed methods research. Example 4 below shows that it is not, 
and that similar integrated, comprehensive inferences can be identified in mono-method 
research. 


Example 4 


González-Ocampo and Castelló (2018) studied the role of writing in doctoral training. Their 
study involved investigation of supervisors’ perspectives on doctoral writing and answered three 
related research questions concerning the role attributed to writing, writing support offered, 
and the relationship between the role of writing and the support offered. Gonzalez-Ocampo 
and Castelló (2018) used one method of data collection, which was a written survey with 
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open-ended questions, and one method of data analysis, which was content analysis using Miles 
and Huberman (1994). Their sample consisted of 61 supervisors in the social sciences and 
humanities at several Spanish universities. One of their conclusions was that 


in some cases, the role attributed to writing was not consistent with the writing sup- 
port offered to students. This result may be related to the lack of awareness of many 
supervisors regarding how writing is supervised. 


(p. 398) 


The structure of this conclusion is displayed in Table 4.4. 

Example 4 is a mono-method study that makes use of a survey with open questions analyzed 
through content analysis. It therefore can contain only instances of mono-method integration, 
in this case, contradiction I and explanation II. Similar processes to those seen in the mixed 
methods studies are visible here: a comprehensive inference is built, in this case, through a 
mono-method contradiction, followed by mono-method explanation. The final, comprehen- 
sive inference (in González-Ocampo & Castelló, 2018, p. 398), of which Example 4 is only a 
part, is much more complex and has a complexity similar to that of mixed methods Examples 
2 and 3. 


The contradiction explanation pattern of integration 


Having considered instances of integration separately, it is now time to uncover the main 
pattern of developing integration throughout a mixed methods study. Figure 4.3 shows that 
in the course of all four studies, one or more contradictions arise, which are subsequently 
explained. In Example 1, the contradiction is that tutors prefer learner-oriented beliefs 
but show a more traditional approach to teaching in the classroom. This contradiction is 


Table 4.4 Development of a Comprehensive Inference in Example 4 


Obtaining Claims Integrating Claims 
Methods Obtained Claims Integrated Claims Connection 
I. Mono-method (qualitative) 
Survey with. — [1] Supervisors attributed a IC [3] The role attributed Contradiction 
open specific role to writing. } to writing was not 
questions — [2] Supervisors offered writing consistent with the 
support to students. writing support offered to 
students. 
II. Mono-method (qualitative) Comprehensive inference 
Survey with. — [3] The role attributed to IC [5] The role attributed to Explanation 
open writing was not consistent writing was not consistent 
questions with the writing support with the writing support 
offered to students. } offered to students. This 
> [4] Supervisors showed a lack result may be related to 
of awareness of how writing the lack of awareness of 
is supervised. many supervisors regarding 


how writing is supervised. 


Note: IC = integrated claim. 
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quan vs. qual 
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I. contradiction qual > quan 
+ N| Ill. explanation 
within qual a 
Il. juxtaposition 
Example 2 
qual + quan qual + quan qual + quan within qual 
|. confirmation |—>| Il. confirmation | Ill. confirmation 4 IV. contradiction qual > mixed 
+ VI. explanation 
qual > qual 
V. explanation 
Example 3 
quan vs. previous qual > quan 
I. contradiction — Il. explanation 
qual > quan 
IV. explanation qual > quan 
within quan a + Nj VI. explanation 
lll. contradiction) N V. explanation A 
qual + reasoning > quan 
Example 4 
within qual qual > qual 
I. contradiction — Il. explanation 


Figure 4.3 The Contradiction— Explanation Pattern of Integration in the Four Example Studies 


Note: > = Explains; qual = Qualitative methods of data collection and data analysis; quan = Quantitative methods 
of data collection and data analysis. 


explained via three factors: the way in which problem-based learning is embedded in the 


curriculum, the confidence teachers have in the self-directed capabilities of students, and 


the self-confidence of teachers regarding their own facilitation skills. In Example 2, after 


various confirmations of the playhouse as a key resource for the children, its positive role 


is contradicted by the statement that the playhouse is also a source of tension. An expla- 


nation for this negative aspect is that the playhouse is too small. Example 3 contains two 
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contradictions, each followed by an explanation. A first contradiction is that, contrary to 
previous studies, providing textbooks did not raise children’s test scores in rural Kenya. 
This contradiction is explained by the fact that most of the children in the Kenyan pri- 
mary schools could not read their textbooks. A second contradiction exists between these 
children and the few children with high pretest scores. For the children with high pretest 
scores, providing textbooks did have an effect on their test scores. The explanation is that, 
most likely, these children were able to read their textbooks. In mono-method Example 4, 
there is a contradiction between the role teachers attributed to writing and the support that 
they offered to students. This contradiction is explained by the fact that many supervisors 
lacked awareness of how writing is supervised. 

Figure 4.3 shows that contradictions arise through diverse combinations of methods. There 
are contradictions between claims from quantitative and qualitative research (Example 1.1), 
within qualitative research (Example 2.IV and 4.1), within quantitative research (Example 3.IIT), 
or with previous research (Example 3.1). The explanation is always based on qualitative research, 
possibly combined with reasoning (Example 3.V). 

In all four examples, the contradictions steer research in a specific direction. On the basis of 
the contradiction, the search for an explanation begins, and these explanations play an impor- 
tant role in the study, which is visible in the meta-inference. Even the “tiny” contradiction and 
its explanation in Example 2 (“the playhouse was a source of tension because it was too small") 
played an important role in the decision to allow the children to build their own temporary 


structures. 


Discussion 


A shift from study-level to individual instances of integration 


This chapter has moved the level of describing integration away from the study as a whole 
(the final meta-inference integrating all claims at once in Figure 4.1) to the level of individual 
instances of integration (the various instances of integration in Figure 4.2). This resonates with 
Maxwell and Loomis’s (2003) statement that “uncovering the actual integration of qualitative 
and quantitative approaches in any particular study is a considerably more complex undertaking 
than simply classifying the study into a particular category on the basis of a few broad dimensions 
or characteristics” (p. 256; emphasis added). Guest (2013) suggested classifying “points of inter- 
face” instead of studies. This chapter can be viewed as an elaboration of this idea. 

As a result, this chapter has uncovered some forms of integration that remain hidden when 
individual instances of claim integration are not considered. First, it identified two forms of 
mono-method integration. It has shown that an explanation can be found within one method 
(e.g., within one interview) and that a subgroup analysis is a form of quantitative mono-method 
integration that might result in two different claims for two different groups. Second, it has 
shown that integration may involve claims from previous research or claims that are the result of 
reasoning without data collection. 


Challenging the uniqueness of the meta-inference 


As this chapter has shown, integration can also be achieved through mono-method research, 
and a mono-method study can also contain an integrated comprehensive inference. This raises 
the question of whether the characteristics ascribed to the meta-inference are unique to mixed 
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methods research. Again, the answer is that these characteristics can also be found in mono- 
method comprehensive inferences. 

The meta-inference in mixed methods research “articulates an understanding of a phenom- 
enon that is not contained in its separate parts” (Bazeley & Kemp, 2012, p. 61); it is a “negoti- 
ated account” (Bryman, 2007, p. 21). This applies to mono-method Example 4 as well. In the 
mono-method example, the contradiction arises by combining the role that supervisors attach 
to writing with the support that supervisors offer. Without one of these two parts, there would 
be no contradiction. Similarly, the explanation is developed by combining information about 
the role of writing and the support provided with information from the interviews. Without 
one of these three parts, there would be no explanation. 

The meta-inference is “rarely a conclusion that could have been predicted from the litera- 
ture” (Creamer, 2018, p. 110). Again, this applies to the mono-method Example 4 as well. It 
is highly unlikely that previous studies had shown that supervisors offer support in writing that 
differs from their view on writing because they are not aware of the role that writing plays in 
supervision. If this were true, the researchers likely would have used a different design to test this 
existing claim, and they would not have come to the conclusion that the contradiction “may be 
related” to a lack of awareness. 

Also not unique to mixed methods research is the process through which integration of 
claims is achieved, a process described as “purposeful conversations . . . and integration,” “a 
cognitive process, conducted in dialogue by an inquiry team and in internal dialogue by a sole 
inquirer” (Greene & Hall, 2010, pp. 125-126), and “a meaningful two-way exchange of infor- 
mation and inferences” (Bazeley, 2018, p. 7). This cognitive process, purposeful conversation, 
or meaningful exchange is visible in the mono-method example: two claims of the study enter 
into a dialogue, this dialogue results in a contradiction, and a third claim continues the dialogue 
by proposing an explanation. 


Integration is integration of claims 


In this chapter, a research process of successive instances of claim integration has replaced the 
common idea in the literature that methods, data, and findings are mixed first and claims are 
integrated next (Figure 4.1). Rather than integration of methods and findings, this chapter fore- 
grounds integration of claims. Claim integration is described as a process that occurs in steps, 
through several instances of integration, rather than all at once (Figure 4.2). Each individual 
instance of integration may or may not involve the mixing of methods. 

This chapter is not in conflict with how the mixed methods research process as a whole is 
described in the methodological literature, as in, for example, “interaction between ideas and 
evidence results in a progressive refinement of the case conceived as a theoretical construct” 
(Ragin, 1992, p. 9) or “a meaningful two-way exchange of information and inferences” (Baze- 
ley, 2018, p. 7). These ideas are compatible with this chapter’s Figure 4.2 but not with Fig- 
ure 4.1, in which claims are integrated all at once. 

Figure 4.2 brings another aspect of integration to the fore: whether an integrated claim is 
the meta-inference ultimately depends on where the study ends. As a study continues, more 
and more claims are integrated. The integration of claims could continue forever, and it is the 
researcher who decides where to stop. Where this happens might differ from the design plan. 
A researcher may, for instance, have planned to proceed until IC [5], but then decide to include 
and to integrate Claim [6], which turned up unexpectedly. Thus, Figure 4.2 shows that both 
the integration process and the mixed methods design process are emergent. 
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The contradiction—explanation pattern as the essence of mixed 
methods research 


This chapter considers the contradiction—explanation pattern as the essence of mixed methods 
research. Rather than one of five possible purposes of mixed methods research (as in Greene 
et al., 1989) or one of 16 (as in Bryman, 2006), this chapter considers contradiction and expla- 
nation to be the characteristics of mixed methods research and, for that matter, much of mono- 
method research. Perhaps radically, Example 2 is not considered a genuine mixed methods study 
until a contradiction shows up, leading to a nuanced conclusion. Thus, although Example 2 
begins with various instances of confirmation, obtained through integrating claims obtained 
from different methods, it has not utilized the potential of mixed methods research until the con- 
tradiction appears. A further development of this idea can be found in Schoonenboom (2019). 

In itself, this pattern of contradiction and explanation is not new. It was described under 
the name of abduction by philosopher Charles Sanders Pierce long ago (Hoffmann, 1999) and 
has been recognized as very important in mixed methods research (Morgan, 2007; Wheeldon, 
2010). This chapter has shown how abduction can be linked to integration. 


Quality in mixed methods research 


If the development of the meta-inference is an emergent process, then what counts as quality 
in mixed methods research should be redefined. According to Cartwright and Hardie (2012), 
the quality of the meta-inference depends on the quality of its premises, which is in line with 
the representation in Figure 4.1: the quality of the meta-inference depends on the quality of its 
constituting claims and cannot be stronger than its weakest claim. Conversely, and inspired by 
Greene (2007, pp. 166—167), this chapter offers the following general principle: “The quality 
of a mixed methods study depends on the extent to which the available data are used to further 
the claims involved.” The quality of a mixed methods study is most visible in, but not identical 
to, the number of instances of integration (Figure 4.2). 

In Example 2, I to III had confirmed the claim that “the playhouse is a key resource for the 
children.” Yet, the contradiction in IV and its subsequent explanation V (“the playhouse is too 
small") modified and furthered this claim and changed the decision on how to develop the out- 
door environment of the preschool. Had the authors stopped at III, the quality of their study, 
according to the definition of quality in this chapter, would have been lower. Similarly, in Exam- 
ple 3, the contradiction in III furthered the development of the meta-inference because it revealed 
a contradiction between children with high pretest scores and children with intermediate or low 
pretest scores. Had the researchers stopped at II, they would have found an explanation for the 
non-occurrence ofan effect (the average child cannot read the textbook"), but not for the differ- 
ence between children with high pretest scores and those with intermediate or low pretest scores. 

In mixed methods research, a contradiction need not necessarily be resolved (Mol, 2002; 
Uprichard & Dawney, 2019). The identification of a contradiction is important in itself. It 
furthers a claim by showing that a simple claim, and therefore the phenomenon that this claim 
describes, is more complex than had initially been thought. 

Thus, rather than on the robustness of the meta-inference, the focus of this chapter is on 
furthering claims. In this approach, claims in mixed methods research may be based on thin evi- 
dence. Sometimes they may be interesting ideas for further research rather than robust findings. 
This does not make these claims less valuable, provided that researchers acknowledge the status of 
their claims as robust findings, interesting ideas, or somewhere in between. Researchers should 
not refrain from formulating interesting ideas that arise from their data, even if the evidence is 


68 


Developing the mixed methods meta-inference 


not robust. The dictum attributed to John Tukey is still true: “An approximate answer to the 
right problem is worth a good deal more than an exact answer to an approximate problem.” 
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